The present invention relates generally to image tracking devices. More specifically, the present invention relates to image tracking devices including an array of light-sensitive elements disposed in a non-rectilinear geometry for tracking an image incident on the array.
Articles and publications set forth herein are presented for the information contained therein: none of the information is admitted to be statutory xe2x80x9cprior artxe2x80x9d and we reserve the right to establish prior inventorship with respect to any such information.
Image sensor arrays are commonplace in modern electronic devices such as displacement sensors, digital cameras, copiers, scanners, facsimile machines, and camcorders, for example. An image sensor array can be a one-dimensional array having a row of light-sensitive elements, or the array can be a two-dimensional array in which there are a plurality of light-sensitive elements disposed in rows and columns of the array. In either case, the array is laid out in a Cartesian geometry (rectilinear geometry) with the light-sensitive elements arranged in an orderly and regular pattern throughout the array. Moreover, the light-sensitive elements are structurally identical to one another and are merely replicated throughout the array. Resulting is a fill-factor that is constant throughout the array. A fill-factor is the ratio of the active light sensing area of a light-sensitive element to the full physical area of the array element.
Image sensor arrays in modern imaging devices such as digital cameras and camcorders, for example, are laid out in a rectilinear geometry. The rectilinear geometry is dictated by semiconductor layout design rules that traditionally require orthogonally arranged circuit elements in order to facilitate semiconductor fabrication. A charge-coupled device (CCD) and a CMOS active pixel sensor are exemplary image sensor arrays that are laid out in a rectilinear geometry.
An optical system that includes one or more lenses can be used to focus an image onto the sensor array. The image can have geometrical distortions such as pincushion distortion, or barrel distortion that are produced by the optical system or by curvature of the object surface being imaged. Additionally, the image can have other field distortions including non-uniform field illumination and non-uniform image resolution. Reducing distortions traditionally requires a lens designer to incorporate additional lens elements, complex lens surface shapes (aspheric lenses), apertures, different or additional glass types, or even filters.
Because the rectilinear geometry of the sensor array has been an accepted design constraint, the burden of correcting the distortions has been placed largely on the optical system. Therefore, as a result, the design variables (the degrees of design freedom) available to the lens designer have been limited to those that can be addressed by changes to the optical system only. However, correcting optical distortions has come at a price, namely, increased cost, weight, size, and complexity of the optical system. Generally, it is desirable to keep the cost of digital imaging devices as low as possible, particularly for products aimed at a consumer mass market. Furthermore, some digital imaging devices are designed to be portable, adding additional design constraints of low weight and small size.
Consequently, there is a need to increase the number of design variables and thereby the degrees of design freedom available to the optics and sensor systems designer. Because geometric distortions are non-rectilinear, one way to increase the number of design variables or degrees of design freedom in the lens and sensor system is to eliminate the design constraint of rectilinear geometry in the sensor. A sensor array having a non-rectilinear geometry can be used to correct geometric distortions in images, reducing the design requirements of the lens. Furthermore, the fill-factors of the light-sensitive elements can be varied with field position to address non-uniform field illumination. And array cell size can be varied with field position to match non-uniform image resolution.
A clever use for image sensor arrays is within sensing devices that track motion of an object. A pattern on a surface of an object is imaged onto an image sensor so that motion of the object causes an image of the surface pattern to traverse an image sensing array.
In a rotary encoder, for example, surface or transmission features of an encoder disc manifest patterns in an image at a detector. Rotation of the encoder disc causes the patterns in the image to traverse an image sensor that is disposed in the rotary encoder. For instance, a light source, such as a light-emitting diode (LED), is used to illuminate the encoder disc so that light from the LED is reflected, scattered off of, or transmitted through the encoder disc to form the image that is incident on the image sensor. The image sensor is laid out in a one-dimensional rectilinear geometry. For example, two to four elements comprising the image sensor can be arranged in a one-dimensional layout (in a straight line).
An example of this in the present art is the use of quadrature signals such as used in shaft encoders to sense angular displacement and direction of a rotating shaft that is coupled to the encoder. However, one disadvantage to a one-dimensional layout of the image sensor when using more than two elements is that coordinate transformation algorithms are required to convert data from the image sensor into data representative of the two-dimensional motion of the image, if maximum resolution is desired.
Another disadvantage of the systematic regularity of a conventional rectilinear layout is that high spatial frequencies in the image can get aliased into the lower spatial frequencies of the data from the image sensor if the distance or pitch length between the elements is more than half the period of those higher frequencies in the image. Aliasing is a disadvantage because it leads to incorrect tracking results.
Therefore, there is a need to overcome disadvantages associated with rectilinear image sensors. Non-rectilinear arrays can be laid out with curvilinear rows and/or columns. For example, when trajectories of object motion are constant as in simple rigid-body motion or in laminar flow, either the rows or columns can parallel the trajectories of image features that move across the array, even in the presence of image distortions. Such a non-rectilinear image sensor array can be used to track a certain two-dimensional motion of an image (incident on the array) without the need to apply coordinate transformation algorithms to make corrections in the post-processing of image data from the image sensor. Furthermore, the spacings between adjacent elements comprising an image sensor array, the position of each active photo sensor element within its cell of the array, and the fill-factor of each element of the array can be adjusted by design to frustrate aliasing.
The problems and limitations associated with rectilinear image sensors are addressed by various aspects of the present invention. The problem with geometrical distortions inherent in an optical system is addressed by positioning photosensors in an array having a non-rectilinear geometry. The array can have a shape that compensates for anticipated geometric distortion in an image; for example, the array can have a pincushion geometry to compensate for pincushion distortion in the image. Problems with field-dependent illumination and resolution are addressed by varying the sizes of the array elements and/or by varying the fill-factors of individual elements. Fill-factor is defined as the ratio of light-sensitive area within an array element to the area of the whole array element. For example, in regions of the image where illumination fall-off is present, the active areas of the light-sensitive elements in those regions can be increased to compensate for the illumination fall-off (relative to active area sizes in regions with less or no fall-off). This can be done by increasing fill-factor while holding cell area constant, by increasing cell area while holding fill-factor constant, or by increasing both cell area and fill-factor. In a second example, resolution can be increased by decreasing cell area. Naturally there is a relationship between illumination control and resolution control that sometimes results in requiring a tradeoff to be made.
Additionally, the problems associated with aliasing of the image data from a moving object are addressed by varying the fill-factors and by varying the positions of the light-sensitive elements within the cell elements of the array so that the spacings between the active area elements of the array vary randomly, with some spacings being greater and some less than the average pitch.
For row and column data from an imaging array, it is desirable to minimize or eliminate the need for coordinate transformation algorithms and related circuitry to convert data from row and column dimensional representations of the image back to two-dimensional representations in object coordinates or to coordinates of the object motion. The present invention addresses that need by using an image sensor array that is laid out in a non-rectilinear geometry that can be formed to match the anticipated geometry of the image motion, even in the case of a distorted image.
For a simple example, the image sensors of the present invention can be arranged in a two-dimensional curvilinear geometry such that object features in the two-dimensional image travel along only the rows or only the columns of the array of image sensor elements as the object moves. This arrangement would eliminate the need to perform two-dimensional image correlation algorithms to track motion, as correlations along only the rows or columns respectively would be sufficient. Moreover, in such a simplifying arrangement, the image sensor elements can be arranged in a two-dimensional non-rectilinear array having rows and columns. The two-dimensional image motion along columns can be tracked by electrically processing image data from each of the rows in the column. Alternatively, the two-dimensional image motion along rows can be tracked by electrically processing image data from each of the columns in the row. Row elements in a column can be combined if the image data is relatively constant along the column, and image motion is along the rows. Column elements in a row can be combined if the image data is relatively constant along the rows, and image motion is along the columns.
In general with image motion having components along both rows and columns, electrical processing is required between nearest neighbors both along columns and rows as well as along diagonals. In any case, the non-rectilinear geometry of the image sensing array of the present invention simplifies image tracking by customizing the layout of the rows and columns of the array to conform to the natural motion inherent in the (distorted) image to be tracked, by sizing pixels throughout the array such that increments of motion become a fraction of local pixel size throughout the array. By assuming that motion throughout the field of view is constant in terms of units of local pixel size, motion can be calculated by simply adding nearest neighbor correlations over the whole array. As a result, image tracking is simplified, by permitting nearest-neighbor correlations to be simply added over all elements, and by eliminating coordinate or other geometrical transformations to correct for geometrical image distortions or non-linear motions in an image.
A general discussion of motion tracking with imaging arrays can be found in the following issued U.S. Pat. Nos. 5,578,813; 5,644,139; 5,686,720; 5,729,008; and 5,825,044. Unlike the image sensing array of the present invention, these issued patents teach the use of rectilinear photo-sensor arrays. Furthermore, these patents teach correlation calculations between nearest neighbor cells between successive images, and sum the results over the cells in the array. Summing in this manner yields accurate results only if motion within the image field is constant in magnitude and direction throughout the area of the array, as when there is no geometric image distortion and the motion is that of rigid-body motion within the image field. If geometric image distortion were present, the motion in the neighborhood of each cell would be different from that of more distant cells, requiring that motion be tracked separately for each neighborhood rather than combined by summing. These xe2x80x9cpointxe2x80x9d measures of motion about the array would then have to be xe2x80x9cassembledxe2x80x9d to produce a composite result. Or the xe2x80x9cpointxe2x80x9d measures of motion could be used as such to represent a motion map over the field of view. Or, alternatively, a-priori knowledge of the geometric distortion could be applied to correct the individual motions measured by the correlations of the many cell neighborhoods, and only thereafter the corrected individual cell results could be summed over the array to produce a measure of xe2x80x9crigid-bodyxe2x80x9d motion within the image field.
Broadly, the present invention provides an image tracking device that includes a plurality of light-sensitive elements that are positioned to define an array having at least one row and a plurality of columns arranged in a non-rectilinear geometry. A cell frame is defined by the intersection of the row with one of the columns. Each cell frame has a frame area and one of the light-sensitive elements is disposed in each cell frame. Each of the light-sensitive elements has an active area and is operative to generate an electrical signal indicative of a portion of an image of an object incident on the active area of that light-sensitive element. The ratio of the active area of a light-sensitive element disposed in a cell frame to the frame area of that cell frame defines a fill-factor. A larger fill-factor results in more of the frame area being occupied by the active area of the light-sensitive element disposed in that cell frame.
In one embodiment of the present invention, the light-sensitive element can be a photodiode, a photocell, a P-I-N diode, a photo transistor, a charge-coupled device (CCD), a CMOS active pixel sensor, an amorphous photo sensor, a solar cell, a photovoltaic device, or a photoconductive device.
In another embodiment of the present invention, the array includes at least a portion of a circularly symmetric geometry having a center point of symmetry. In one embodiment the array includes a plurality of rows, each row arranged at a constant radius from the center point of symmetry and pitch between the rows decreases with increasing radial distance from the center point of symmetry, so that the frame area is constant across the array. The center point of symmetry can be substantially coincident with an axis of rotation of an image of an object.
In one embodiment of the present invention, the array has a pincushion geometry so that features on a concave spherical object can be imaged onto the array. In another embodiment the array has a barrel geometry so that features on a convex spherical object can be imaged onto the array.
In one embodiment of the present invention, the geometry of the array is laid out in a pattern that can be a pincushion, a barrel, a keyhole, a trapezoid, or a parallelogram to compensate for geometric distortions in an image incident on the array.
In another embodiment, cell sizes (and respective active areas) are both decreased in regions where higher image resolution is desired and vice versa.
In other embodiments of the present invention, optical distortions and illumination fall-off in an image incident on the array are compensated for by selecting the frame area and/or by selecting the size and/or shape of the active area.
In one embodiment of the present invention, the distance between cells frames in adjacent columns define a column cell pitch, and the distance between cell frames in adjacent rows defines a row cell pitch. The light-sensitive element in each cell frame is positioned or shaped randomly in that cell frame or varied systematically so that aliasing is minimized ( i.e., for images having some spatial periods less than about twice the row cell pitch or less than about twice the column cell pitch). Alternatively, the active area of the light-sensitive element in each cell frame can be randomized so that the shape and/or fill-factor vary randomly throughout the array and aliasing is minimized for images having spatial periods less than about twice the row cell pitch or less than about twice the column cell pitch.
Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present invention.